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In our experience of having worked with financial institutions for over 19 
calendar years, we have seen that application development, maintenance and 
operations teams have a very silo-ed existence. In today's ever more fluid 
market, it is imperative for these teams to be very agile in their response to 
business needs. While development teams are trying to adopt agile and 


devops practices, operations and maintenance teams are sometimes forgotten 





in this transformation journey even though they play a very crucial role in 
Keywords: managing customer facing systems. In this paper, we focus on application 
operations for their simplification, outline the post production challenges 
es faced by the operations teams, assistance required from the development 
Application support teams and need for coordination and harmony between application 
Cloud adoption development and operations teams. We use the example of a financial 
DevOps organization where we are currently helping in simplification of application 
Simplification support processes using two key levers of cloud and devops adoption to 
highlight the path to simplification. We outline the solution levers we used, 
solution tenets, solution approach and considerations. We also share a target 
state model and regulated key functions and activities within the model as 
well as an interim operating model and a target operating model. 


Agility 


Copyright © 2019 Institute of Advanced Engineering and Science. 
All rights reserved. 





Corresponding Author: 


Anshu Premchand 

B.S. Abdur Rahman Crescent Institute of Science and Technology GST Road, 
Vandalur, Chennai 600 048, Tamilnadu, India. 

Email: anshuprem@ gmail.com 








1. INTRODUCTION 

Software service providers and enterprise development teams are increasingly under pressure to 
create, enhance and deliver applications at an ever-faster pace. Agile software development practices and 
devOps help reduce release cycle times but costly and time consuming on-production application errors or 
mistakes still ail developers, systems administrators and IT support teams irrespective of devOps teams 
having been established on the application development side. Application problems commonly arise from 
change of status quo that results in disharmony with upstream and/or downstream systems. The rapid changes 
caused by agile development therefore necessitate solving application support problems very quickly. Also, 
the concept of bring your own device to office requires agility to handle application support systems [1]. The 
existing inter-device collaboration system is generally centralized system [2]. 

One of primary goals of devOps is to get developers more involved in application maintenance and 
operations so they can more effectively resolve application issues in production environments [3]. The other 
important goal is to bring development and testing environments as close to each other as possible. This helps 
because normally, there are many production issues that cannot be replicated in development environments 
because they are so disparate. But if the environments are same or similar, the number of ‘new’ issues 
occurring in production environments can be brought down measurably. Also, if developers do not have 
access to live production applications and servers, troubleshooting issues can become very time consuming. 
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This leads to a backlog of system defects, unhappy customers, and business and overwhelmed development 
and operations teams [4]. With increased adoption of big data, Internet of Things and sensor technology by 
various organization for provisioning smart intelligent services for various application uses, the system 
becomes complex [5]. In certain industries like financial institutions, roles & responsibilities dictate access to 
environments. These are essential for risk & compliance, amongst other things [6]. We worked with a large 
financial institution from the fortune 500 list and helped simplify their application support by agility and 
devOps adoption and delivered quantified benefits between 10-25% of cost to manage applications for 
various technology sets. The lower end of the spectrum saw legacy applications and COTS (common off the 
shelf) product sets and the higher end saw applications written in technologies such as java, Net and newer 
digital technologies. 

In our case with financial institution we mentioned, we used the levers of agility, devops and 
cloud [7]. The delivery managers at the financial organizations had two primary challenges - one, finding 
skilled developers to fill open positions to improve output from the team and two, improving throughput and 
productivity of teams. Developers normally spend a large percentage of their time working on application 
defects and support problems when their time would be better spent on creating new solutions to further the 
cause of business. Using devOps and agile practices, we helped improve development processes, added tools 
and created CI-CD pipelines (continuous integration — continuous delivery) for the financial institution and 
recorded an overall productivity improvement of 6%-15% while maintaining existing headcounts. 
This additional capacity was utilized to deliver new solutions to business. In this paper, we share our 
experience of working with the financial institution on improving & simplifying their application support 
processes. With this example, we explain how common barriers to production application support agility can 
be overcome in a time and cost efficient manner with a DevOps approach supported by agile practice 
adoption, while reducing the cross-functional friction that commonly arises when deployed applications stop 
running or show defects [8]. 

We used a two pronged approach to solve this problem — one, we used devops principles and 
second, we used cloud (private cloud and public cloud). 


2. THE PROPOSED PROCEDURE: SIMPLIFYING AND IMPROVING APPLICATION 
SUPPORT IN PRODUCTION 

We observed at the financial institution that most development teams and individual developers 
spent approximately 20 to 30 percent of their time on application support and defect management. Ideally, 
the development teams can be granted read-only and audited access to infrastructure resources so they can 
get the information they need without exposing the organization to unnecessary risks including (and not 
limited to) financial risks. We helped simplify the operations processes to: 
a) Provide cross-functional devOps visibility 
b) Maintain security, risk management and compliance policies 
c) Control access rights as needed by role 
d) Filter, mask and harness sensitive data 
e) Audit developer access to production systems 
f) Create dashboards for application health monitoring 
g) Generating reports as needed and keeping right stakeholders in loop 

In our example, with simplified processes, development and operations teams were able to report 
increase in collaboration and were able to gain shared visibility to solve production application support 
problems faster and reported lesser defects in production over a period of time. We were able to report 
increased productivity and improved collaboration throughout their development and operations teams by 
breaking down silos and creating a unified view of the enterprise. We will explain the method applied in 
sections that follow. After first and second wave of simplification, we were able to: 
a) Reduce operations and support overhead and costs 
b) Reduce licensing costs 
c) Resolve customer-impacting production issues faster 
d) Improve up-time 
e) Benefit from greater collaboration and knowledge sharing 
f) Get better real-time visibility across various environments and rationalize the number of environments 
g) Successfully manage risk, monitoring and audit requirements through role-based access for developers 

and tracking of all troubleshooting and change activities 

h) Easier migration of applications to cloud (as needed) 
i) Mitigate & manage risks and improve effectiveness of outsourced development and support resources 
j) Scale on demand the development and operations teams by cross training 
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The “Scaling on Demand” worked in two ways: 
1. Developers were able to get a dashboard view of all of their applications and related servers. They were 
able to track the health of the entire application stack. They could, without assistance : 
a) Automatically discover, monitor and restart applications 
b) Centrally view apps from a data center(s) 
c) Manage cloud based applications (if any) 
d) Track changes 
e) See applications, services and scheduled tasks 
f) Access log and configuration files 
g) Query production databases 
h) Monitor application and server health 
i) Get notifications and alerts 
2. Operations teams were able to simplify their workload by: 
a) Giving development teams safe and secure access to production applications and production servers. 
b) Provide cross-functional devOps visibility 
c) Maintain security, risk management and compliance policies 
d) Control access rights 
e) Audit developer access to production systems 
We carefully went through all the requirements of the financial organization and applied a multi- 
pronged approach. We split the requirements in to two service bundles 
a) Bundle one: Traditional IT and Private Cloud 
b) Bundle two: Public Cloud 
The key motivator was to ensure a flawless digital customer journey and the IT-infrastructure 
needed to support this. Modernization, streamlining and consolidation were the key themes in structural 
change at the financial institution we were helping. The financial organization need to drive changes from a 
business perspective & for IT to be successful, a strong cooperation and collaboration with the business was 
necessary. To achieve all these, we proposed adoption of agile and devops practices [9-10]. Table 1 broadly 
lists our recommendations for the financial organization’s business objectives shared with us. 


Table 1. Mapping of Business Objectives to Our Recommendations 








Business Objectives Our Recommendations 

System, service and business availability needs (24/7) Highly resilient infrastructure in all layers on cloud infrastructure 
Capacity fluctuation management Public cloud for reliability and scalability 
Simplified tools & services to manage cost Simplified, orchestrated tool chain 

Reduce size of code base Consider cloud brokerage service model, cognitive automation for 

operational efficiencies 

Simplified integration management Template based integration 
Seamless digital customer journey supported by IT for Leverage cloud to provide diverse set of IaaS and PaaS services 


improved customer satisfaction 





Adopting to the three key themes of structural changes namely Modernization, Streamlining and 
Consolidation, we constructed the target state solution to meet the current and future requirements of the 
financial organization. 

Figure 1 shows the key solution drivers for the entire exercise. The key drivers were divided into 6 
classes which drove one or more key tenets of the solution. These driver classes were cloud, automation, 
digital platforms, governance, security and cost. For each driver class there were sub-focus areas. 
For example, for digital, the sub focus areas included cloud bursting, modern integration services, and micro- 
service enablement. For governance, the sub-focus areas were cloud service brokerage, cloud governance, 
architecture board and competence management for regulations and so on. These drivers helped us arrive that 
the key tenets of the entire proposed solution for simplification of application operations as we have 
described below. The key tenets of the proposed solution were: 

a) Zero disturbance knowledge transition 

b) Cloud assessment enabled with application centric assessment approach 

c) Cloud focused approach for workload migrations 

d) Migration and transformation risks with mitigation 

e) Complete scope of services on IT hosting and operations 

f) | Scalable and Reliable target cloud platform 

g) Focused automation for stable infrastructure 

h) Centralized monitoring with event correlation across platforms, applications and network 
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Cloud governance enabled operating model designed with single point of ownership 
Quality support 

Support improvements through operational process improvements 

Run support for cost efficiencies for both hosting and operational services 

Manage Risks 

Allow customer controlled personalization on all new digital applications 

An ideal team mix with all relevant skills and enabled with cloud governance 
Operational transparency through service management tools and processes 

Digital and autonomous platform to address current and future requirements 
Continuous service improvements through analytics, automation and technology and process driven 
hosting and support enhancements 


e Clou qi Cloud Platform 


Cloud Market Place driven 
* Agile development & operations support 


Cost Efficiency * Application Centric Infrastructure 

* Right Sizing & placement a 

* Operation Efficiencies = 

* System stability Automation 

* Ramp Up / Down AUTOMATION * Focused Automation & Orchestration 








* Infrastructure as Code (laC) 
* Self-heal solution for stability 
* Reduce manual errors 


ie 


* Modern Integration Services 
* Micro-service Enabled 


Solution 
Drivers 





Security 


* Micro segmentation 
Meeting Compliance 

* Private IP to access Public 
Cloud 


Digital platforms 


Governance 

* Cloud Service Brokerage 

* Cloud Governance 

* Architecture board 

* High Competence in regulations 


Figure 1. Key solution drivers 


Broadly, as Table 2 shows, we classified all the current capabilities and/or services available into 


these two bundles to clearly specify which one of these services could be handled using traditional IT and 
private cloud and which ones could be taken to a public cloud solution. Since we were focusing on 
application support, for this simplification exercise, the following services were deemed out of scope: 


a) 
b) 
c) 
d) 


Application Development - Build, Configure, Test 

Application Maintenance - 3rd Level Application Support, Minor enhancements, bug fixes 
Archiving 

Data Center Decommission 


Table 2. Bundle Wise Split of Activities 








Current Capabilities Traditional IT & Private Cloud Public Cloud 
Cross Functional Services Yes Yes 
Security Operation Center Yes Yes 

Application Operations Yes Yes 
Integration Service Yes Yes 
Database Management Yes Yes 
Network Services Yes No 

IaaS (Mainframes, legacy) Yes No 
PaaS (all types of databases) Yes Yes 
IaaS (all types of OS) Yes Yes 
Storage Capacity Yes No 
Server Capacity Yes No 
Management of SaaS No Yes 
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We also analyzed the existing infrastructure and drew observations and then mapped each of these 
observations to inferences that helped define the target operating model (TOM) and target simplified 
application support/operations solution. Some of the important observations and inferences are listed 
in Table 3. 


Table 3. Our Key Observations & Inferences Drawn 








Observations Inferences 
The existing infrastructure had minimal or no Consider moving possible non-sensitive workloads to public cloud, 
footprint on public cloud which would enable higher scalability and flexibility in the target 
infrastructure 
20% of the systems were in physical form Possibility of migrating this 20% infrastructure to virtual form which 


would give 100% virtual infrastructure, which is an excellent enabler for 
digital technologies like Internet of Things (IoT) etc. 
No backup archival solution even though there were _ Possibility of deploying consolidated archival solution with compliance 
multiple application based archival solutions 


available 

No cloud management platform and no self-service Deployment of hybrid cloud platform enabled with user friendly 

portal based provisioning interfaces which help with faster go-to market application deployments 
Multiple versions of operating systems Operating systems standardization while application migration 

Legacy integration platforms Modernization of current integration platforms 

No automated disaster recovery solution Possibility of bringing in DR automation 





3. THE METHOD 

We also followed data-based approach of performing assessment of the current infrastructure and 
workloads to qualify the cloud workloads and understand the current architectural gaps. As a part of the 
solution we proposed to perform a quick assessment for cloud adoption during transition. We wanted to 
assess, design, build and migrate the infrastructure over the period of 6 to 8 months as per the agreed 
migration timeline. Our plan was to divide these tasks into 3 distinct phases: 

a) Phase 1: We would perform discovery and cloud assessment (~ 3 Months) as a part of transition to 
find out the current infrastructure state and qualify the candidates for migration to cloud 
(both private and public cloud). The recommended that the assessment report be analyzed by the 
customer and based on customer’s feedback the target state infrastructure will be designed. 
The target state architecture will get approved by the customer prior to deployment. 

b) Phase 2: In phase 2, our plan was to construct the target infrastructure on enterprise cloud 
infrastructure located in customer’s geographic location. This was done to ensure that data related 
compliance needs were met. Current workloads on incumbent datacenters would be migrated to the 
cloud in a phased manner over the period of 4 to 6 months (excluding assessment). The target 
infrastructure was to be built based on the design accepted by customer, all relevant cloud and 
management tools would be deployed and integrated. Customer application specific service 
catalogues would be created and deployed on recommended ITSM tool and integrated with cloud 
management platform. 

c) Phase 3: In phase 3, we would be providing end to end cloud services for the financial organization 
with strong cloud governance in place. We were to be acting as a cloud service brokerage (CSB) 
provider to ensure that the customer was assured of optimized target infrastructure and ability to 
adhere to market transformation rapidly. 

As shown in Figure 2, we looked at business domain, technology and service delivery as key inputs 
and mapped these to key transformations such as knowledge management system setup, cloud migration, 
standardization & governance, API enabled integration and digital transformation. The figure shows the 
movement from start to end state with timeline moving from 1-ni weeks with initiatives in the timelines for 
various key transformations. This helped us draw a business centric migration and transformation approach 
with timelines where we could target and track each transformation from start to end state. Business Centric 
approach was applied at all layers of the infrastructure to bring in the complete benefits to the financial 
organization. Data center was treated as scalable platform which would bring speed and agility in business 
application deployments, servers were treated as group of business components hence failure of each 
component was mapped with failure of business functionality and hence the architecture would be realigned 
based on the business functionality criticality. 
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Figure 2. Business centric migration & transformation approach 


SOLUTION CONSIDERATIONS 


The following were the key solution considerations we had taken into account while constructing 


the target infrastructure: 


a) 


b) 
c) 
d) 
e) 
f) 


g) 
h) 
i) 
aD) 
k) 


) 


m) 


n) 


Two cloud infrastructure locations within the country limits as two availability zones to ensure data 
related compliance requirements are met. One of the locations to also act as primary disaster 
recovery location 
We recommended a serious look at “as a service” models to the financial organization to bring down 
costs, which included IaaS and PaaS services 
We requested considering to host 50% split of workloads in between two locations to provide 
active-active configuration of the workloads 
Storage tiers and sizing had been considered based on current data provided by the financial 
organization, to be validated during due diligence stage 
Backup considerations were based on current requirements given by the customer, this would be 
validated during due diligence 
Archival solution was proposed as optional solution 
Target infrastructure will be enabled with maximum automation possible 
We recommended considering self-heal solutions be considered as a part of the target infrastructure 
Consolidated Integration Services platform solution had been considered 
Target infrastructure is built to ensure cloud service brokerage model delivery 
The target infrastructure monitoring and management solution had been constructed to handle 
hybrid cloud platform and legacy devices 
Business centric migration move groups would be created during transformation 
Target infrastructure had features enabled for future technology transformations such as Server-less 
Computing, Digital, and Internet of Things etc. 
Cloud LAN hardware and software were sized suitably for server workloads and network 
components which include: 

1) LAN Switching Fabric 


2) Perimeter & Internal Firewall 

3) Intrusion Detection and Prevention at important Ingress/ Egress Zones 

4)  External/DMZ (demilitarized zone) and Internal Load Balancer 

5) Remote Client Terminal Connection to access financial organization’s resources 
6) Site to site VPN (virtual private network) 

7) Support functions for IP Addresses, DHCP, DNS 
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5. RESULTS AND DISCUSSION: INFRASTRUCTURE SECURITY, MONITORING AND 
MANAGEMENT 

We recommended that cyber security be updated and transitioned to SIEM (Security Information & 
Event Management) Platform and enterprise vulnerability management, identity access management, 
public key infrastructure and other platform L1/L2/L3 security services be spruced up in different business 
verticals. We recommended the following services be delivered as part of security operations center for the 
financial organization: 

a) Security Information and Event Management 
b) Vulnerability Assessment 

c) Penetration Testing 

d) Identity and Access Management 

e) Public Key Infrastructure 

We understood that the financial organization’s long term strategy was to move enterprise 

workloads to public cloud. Therefore, our recommendations were the following: 
a) Unified infrastructure and application monitoring 
b) Continuous discovery and automated relationship maps 
c) Financial analytics on public cloud expenses, cost and resource optimizations 
d) Unified dashboards & reports for cloud operations 
We will detail the security, monitoring and management solution in our next paper. 
Application Operations Rationalization using DevOps for Simplified Management: 

We recommended that the financial organization strategically move towards a devOps model for 
their application operations, maintenance and development work. The main driver for this change was a need 
to improve time-to-market for new application functionality or other code changes. 

We proposed DevOps as a Service (DaaS) model which povides all the benefits of devOps as a 
managed cloud service. Based on the DevOps Maturity Assessment of the financial organization’s current 
state, we will be able to create a time-bound, milestone based plan to automate as much as possible and bring 
the entire continuous deployment pipeline (CD) onto the public cloud. This would enable the financial 
organization to focus on other areas without having to worry about the devOps stack management and IT4IT 
work that will come under the "Managed DaaS" umbrella. 

During Due Diligence phase, a feasibility analysis would be done jointly with the financial 
organization to understand all of their specific needs and provide customized solutions leveraging public 
cloud for devOps adoption keeping in mind various other factors like cost, performance of tools and 
orchestration needs. From the state of technology operations of the financial organization, we understood that 
one-pass move to the target state we wanted to reach would be impossible. We would have to target to reach 
an intermediate operating model where all simplification themes were underway. This would also help us 
keep track of progress and course correct as needed. 

As collaboration is key for a successful devOps model, we recommended that Application 
Operations (AO) team will perform the following activities: 

a) Provide clear definition of processes and responsibilities across the devOps organizations (lines of 
business within the financial organization) and the application operations organization 

b) Roadmap for the implementation covering all relevant areas such as suggesting a stepwise approach 

c) With the sequence of implementation and pace for roll-out — such as area to start with and how to 
gradually scale up 

d) Selection and implementation of tools supporting devOps 

e) Actual implementation 

f) | Orchestration across the tool-chain 

Figure 3 shows the target operating model for the application support services of the financial 
organization. We are on track to attain the same. The vertical units refer to shared functions across the 
organization like audit and regulatory, IT4IT teams, risk and compliance management etc. The horizontal 
units on top are the business groups, business users and service desk that consume the services below them. 
Cloud management platform works on top of the business aware command center which automation layer 
that works on top of the other platforms such as legacy platforms, application operations and so on. The final 
horizontal layer represents the enterprise cloud and public cloud service providers. 
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Figure 3. Target operating model 


We also requested support from the financial organization and its application development & 
maintenance (AD/AM) teams. The following dependencies on the application development & maintenance 
(AD/AM) teams was foreseen: 

a) The selection of tools (e.g., deployment tools) shall be aligned with the financial organization’s 
initiatives for automated deployment on test and production. 

b) Definition of when different applications/teams are ready for transformation 

c) Availability (time and focus) of development team to absorb knowledge and practice the new 
capabilities 

d) Dependency on other application development teams for release coordination 


6. CONCLUSION 

Simplification using cloud and devops adoption cannot happen until operations and infrastructure 
organizations are made a part of the simplification process and rationalized. These groups need to be 
disrupted and transformed into an effective devops model to ensure business needs of today’s digital world 
are met. Agile & devops practices in culture, automation, lean, measurement and collaboration aspects are 
great enablers for simplification of operations and infrastructure organizations and cloud adoption (private/ 
public/hybrid) is an excellent enabler of the simplification itself. These levers lead to quantifiable benefits. In 
a short while, we will be able to write another paper to share the quantified benefits accrued to the financial 
organization referred in this paper due to the simplification of application support services as we have 
outlined here. Operations and infrastructure teams may also need to re-skill, cross skill and align to the new 
way of working and also use cohesive business centric key process indicators. The post production 
challenges and assistance needed from development teams we have discussed will find resonance in many 
similar organizations. The solution tenets, solution approach, interim and target operating model can act as a 
guiding force for organizations with similar issues. Through the transformational approach we have outlined 
in this paper the application operations teams will be in the driver’s seat in the organization’s journey 
towards simplification and delivering greater business value to all stakeholders. 
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